Human Daily Activities Indexing in Videos from Wearable Cameras for Monitoring of Patients with Dementia Diseases

Our research focuses on analysing human activities according to a known behaviorist scenario, in case of noisy and high dimensional collected data. The data come from the monitoring of patients with dementia diseases by wearable cameras. We define a structural model of video recordings based on a Hidden Markov Model. New spatio-temporal features, color features and localization features are proposed as observations. First results in recognition of activities are promising

The IMMED Project: Wearable Video Monitoring of People with Age Dementia

International audienceIn this paper, we describe a new application for multimedia indexing, using a system that monitors the instrumental activities of daily living to assess the cognitive decline caused by dementia. The system is composed of a wearable camera device designed to capture audio and video data of the instrumental activities of a patient, which is leveraged with multimedia indexing techniques in order to allow medical specialists to analyze several hour long observation shots efficiently

Hierarchical Hidden Markov Model in Detecting Activities of Daily Living in Wearable Videos for Studies of Dementia

International audienceThis paper presents a method for indexing activities of daily living in videos obtained from wearable cameras. In the context of dementia diagnosis by doctors, the videos are recorded at patients' houses and later visualized by the medical practitioners. The videos may last up to two hours, therefore a tool for an efficient navigation in terms of activities of interest is crucial for the doctors. The specific recording mode provides video data which are really difficult, being a single sequence shot where strong motion and sharp lighting changes often appear. Our work introduces an automatic motion based segmentation of the video and a video structuring approach in terms of activities by a hierarchical two-level Hidden Markov Model. We define our description space over motion and visual characteristics of video and audio channels. Experiments on real data obtained from the recording at home of several patients show the difficulty of the task and the promising results of our approach

Troubles de l'orientation spatiale et troubles visuo-constructifs dans le vieillissement normal et la Maladie d'Alzheimer

Nous avons étudié les déficits d'orientation spatiale et visuo-constructifs dans le vieillissement normal et dans la maladie d'Alzheimer (MA). Dans un premier temps, nous avons utilisé un labyrinthe à échelle humaine pour explorer les déficits d'apprentissage de trajets dans la MA. Ces patients sont en difficulté face à des situations de planification des déplacements et dans des tâches nécessitant la manipulation mentale de représentations mais sont capables de se déplacer par guidance. Dans un deuxième temps, nous avons étudié les troubles visuo-constructifs dans le dessin du cube et de l'horloge, depuis les données des cohortes en population Paquid et 3 Cités. Les résultats montrent des erreurs même chez les sujets âgés non déments dans les deux dessins ainsi que l'existence d'erreurs plus fréquemment commises par les sujets déments. Connaître les processus cognitifs engagés dans ces tâches est nécessaire pour développer des stratégies de compensation.We studied impairments in spatial orientation and in visuo-constructive abilities in normal ageing and Alzheimer's disease (AD). Firstly, thanks to experimentations in a human-sized maze we investigated navigation deficits in AD. The patients were impaired in planning and unable to acquire a mental representation of their environment. Consequently, they were impaired in tasks requiring a mental manipulation but they could walk following a guided path. Secondly, we studied visuo-constructive impairments appearing in a cube and a clock drawing. Data were collected among two French population-based cohorts : Paquid and 3 City studies. Results evidence the occurence of some types of errors in non demented patients and some other types occuring more frequently in dementia. Knowing cognitive processes involved in these domains is necessary to develop compensatory strategies.BORDEAUX2-BU Santé (330632101) / SudocSudocFranceF

Detection of moving foreground objects in videos with strong camera motion

International audienceIn this paper, we propose a novel method for moving foreground object extraction in sequences taken by a wearable camera, with strong motion. We use camera motion compensated frame differencing, enhanced with a novel kernel-based estimation of the probability density function of background pixels. The probability density functions are used for filtering false foreground pixels on the motion compensated difference frame. The estimation is based on a limited number of measurements; therefore, we introduce a special, spatio-temporal sample point selection and an adaptive thresholding method to deal with this challenge. Foreground objects are built with the DBSCAN algorithm from detected foreground pixels

Human Daily Activities Indexing in Videos from Wearable Cameras for Monitoring of Patients with Dementia Diseases

International audienceOur research focuses on analysing human activities according to a known behaviorist scenario, in case of noisy and high dimensional collected data. The data come from the monitoring of patients with dementia diseases by wearable cameras. We define a structural model of video recordings based on a Hidden Markov Model. New spatio-temporal features, color features and localization features are proposed as observations. First results in recognition of activities are promising

Visual saliency maps for studies of behavior of patients with neurodegenerative diseases: Observer's versus Actor's points of view

International audienceWe are interested in finding the relation between the visual saliency maps of the viewer of visual content and the actors (person executing the actions) in the context of studies of neurodegenerative diseases such as Alzheimer's disease. From results of eye-trackers worn by the actors and used when recording observers, and on the basis of hand-eye interactions from motor control studies we established a time shift between actor's and viewer's saliency maps. This time shift corresponds to the latency of hand-eye interaction. The method is based on adequate normalization of saliency maps and computation of similarity metrics for pixel based saliency. This finding gives good perspectives for automatic prediction of a normal actor saliency map from observer saliency map

List of scientific publications

International audienceThis paper presents a method for indexing human ac- tivities in videos captured from a wearable camera being worn by patients, for studies of progression of the dementia diseases. Our method aims to produce indexes to facilitate the navigation throughout the individual video recordings, which could help doctors search for early signs of the dis- ease in the activities of daily living. The recorded videos have strong motion and sharp lighting changes, inducing noise for the analysis. The proposed approach is based on a two steps analysis. First, we propose a new approach to segment this type of video, based on apparent motion. Each segment is characterized by two original motion de- scriptors, as well as color, and audio descriptors. Second, a Hidden-Markov Model formulation is used to merge the multimodal audio and video features, and classify the test segments. Experiments show the good properties of the ap- proach on real data

Strategies for multiple feature fusion with Hierarchical HMM: Application to activity recognition from wearable audiovisual sensors

International audienceIn this paper, we further develop the research on recognition of activities, in videos recorded with wearable cameras, with Hierarchical Hidden Markov Model classifiers. The visual scenes being of a strong complexity in terms of motion and visual content, good performances have been obtained using multiple visual and audio cues. The adequate fusion of features from physically different description spaces remains an open issue not only for this particular task, but in multiple problems of pattern recognition. A study of optimal fusion strategies in the HMM framework is proposed. We design and exploit early, intermediate and late fusions with emitting states in the H-HMM. The results obtained on a corpus recorded by healthy volunteers and patients in a longitudinal dementia study allow choosing optimal fusion strategies as a function of target activity

Egocentric vision IT technologies for Alzheimer disease assessment and studies

Abstract – Egocentric vision technology consists in capturing the actions of persons from their own visual point of view using wearable camera sensors. We apply this new paradigm to instrumental activities monitoring with the objective of providing new tools for the clinical evaluation of the impact of the disease on persons with dementia. In this paper, we introduce the current state of the development of this technology and focus on two technology modules: automatic location estimation and visual saliency estimation for content interpretation